What is the right course of action here?

You answered correctly. The "yield" (give way) sign means you must let all cross traffic pass before you proceed. The yellow vehicle is on the priority road and has the right of way, so you must let it go first. Only after the yellow vehicle has passed can you turn. This rule helps prevent accidents at intersections where not all drivers have equal priority. Always pay attention to yield signs and give way to vehicles on priority roads.